Local optimization for curvy brush stroke synthesis

ABSTRACT

Systems and methods provide for on the fly creation of curvy, digital brush strokes using incremental, local optimization. Samples from a user&#39;s input stroke are detected and matched with exemplar brush stroke segments as the user proceeds to provide input. For each set of samples, a temporary segment is generated and displayed for the user, and the temporary segment is later replaced by a permanent segment as subsequent samples sets are matched. Additionally, this optimization allows for updated parameterization in corner regions to provide a more realistic curve in the digital brush stroke. Specifically, intersecting ribs in the corners may be collapsed to prevent the rendering of artifacts. Additionally, and corner structures may be inserted in a break in a corner structure. These corner structures may be extensions of samples around the break and may correct distortion that results from the rib collapsing.

BACKGROUND

Many applications have presented users with the option to paint adigital picture by making digital brush strokes. At its most basic form,rendering a digital brush stroke involves application of a user inputstroke, which may be indicated by movement of a pointing device acrossan electronic canvas, to a particular stroke synthesis technology, suchas an algorithm, to obtain an output of a digital brush stroke followingthe input stroke. Unlike with traditional, physical painting, thereexists some lag time between the user's input and the rendering of thedigital brush stroke. For the painting experience to be interactive,however, it is desirable to reduce this lag time as much as possible tohave the digital brush stroke displayed almost immediately as the inputstroke moves across the electronic canvas. A digital brush strokesynthesized in near real time, or “on the fly,” provides a user withmore control.

This lag time during rendering is particularly prominent with morecomplex stroke synthesis technologies, such as data-driven approachesthat use real brush stroke samples to recreate a natural appearingdigital brush stroke. While these complex approaches have the advantageof providing output closely resembling physical painting media, theytend to require more processing to render the digital brush stroke,thereby increasing the lag time for the output to be displayed. Further,data-driven models also experience difficulties in parameterizing cornerregions of a curve in a way to prevent artifacts, texture stretching,and shape distortion, all of which limit the natural appearance of acurvy, digital brush stroke.

SUMMARY

Aspects of the present invention relate to, among other things,continuous pattern matching to generate a digital brush stroke byutilizing local, incremental optimization to provide immediate or nearimmediate visual feedback. The disclosed continuous pattern matchingdetects samples from a user's input stroke and matches those sampleswith exemplar brush stroke segments as the user proceeds to provideinput. For each set of samples, a temporary segment is generated anddisplayed for the user, and the temporary segment is later replaced by apermanent segment as subsequent samples sets are matched. In this way,the permanent segment may have a superior appearance because it is basedon more samples, but the user does not have to wait to receive visualfeedback until a desired brush stroke is fully inputted by the user.Instead, pattern matching techniques are used at an incremental level tosynthesize a digital brush stroke “on the fly.”

Embodiments of the present invention also relate to generating realisticcorner regions of digital brush strokes on the fly. The usualparameterization of a brush stroke for pattern matching can lead toartifacts in or distortion of the corner regions of the digital brushstroke. To avoid artifacts, embodiments collapse intersecting ribs thatdefine the parameters of a brush stroke such that the endpoints of theribs within a corner region meet at a single intersect point.Additionally, as ribs are collapsed, the rib lengths are compared tothreshold values to detect distortion in the corner regions. Whendistortion is detected, corner structures are inserted, and theresulting brush strokes are generated using the corner structures.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 depicts a block diagram illustrating an example operatingenvironment in accordance with some implementations of the presentdisclosure;

FIG. 2 depicts parameterization of a digital brush stroke in accordancewith implementations of the present disclosure;

FIG. 3 depicts a flow diagram illustrating an example method ofcontinuous pattern matching to generate a digital brush stroke inaccordance with some implementations of the present disclosure;

FIGS. 4A-B depict a curved, digital brush stroke created by traditionalmethods and a curved, digital brush stroke created in accordance withimplementations of the present disclosure, respectively;

FIGS. 5A-C depict parameterization of a curved, digital brush stroke inaccordance with some implementations of the present disclosure;

FIG. 6 depicts a flow diagram illustrating an example method ofgenerating a curved, digital brush stroke in accordance with someimplementations of the present disclosure;

FIGS. 7A-B depict parameterization of a curved, digital brush stroke inaccordance with further implementations of the present disclosure;

FIG. 8 depicts a flow diagram illustrating an example method ofgenerating a curved, digital brush stroke in accordance with furtherimplementations of the present disclosure;

FIG. 9 depicts parameterization of a curved, digital brush stroke inaccordance with further implementations of the present disclosure; and

FIG. 10 depicts a block diagram of an exemplary computing environmentsuitable for use in implementations of the present disclosure.

DETAILED DESCRIPTION

The subject matter of the present invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Some methods for creating digital brush strokes involve matching samplestaken from a user's input stroke, such as a line drawn with a stylus, tostored brush strokes. Generally, more samples that are available formatching provides for greater accuracy in matching the samples,producing a more natural appearance for the digital brush stroke. Forthis reason, global optimization is traditionally used with matchingtechniques, particularly complex, data-driven techniques. As usedherein, “optimization” refers to matching samples to exemplar brushstroke segments and synthesizing a resulting digital brush strokesegment. As used herein, “exemplar brush stroke segments” refer topreviously stored brush stroke segments that are used as examples forgenerating new brush strokes.

While local optimization may synthesize a brush stroke incrementallybased on small portions of a desired brush stroke path, globaloptimization synthesizes digital brush strokes based on all orsubstantially all of a desired brush stroke path. As such, globaloptimization requires a user to wait until the user indicates a largeamount of the desired brush stroke path and, in some instance, theentire path, before matching and brush stroke synthesis occurs. Waitingfor more samples to use for matching and synthesis, however, creates avisual feedback delay that users may find undesirable or evenintolerable. This visual delay may be perceived by the presence of emptyspace between the last rendered digital brush stroke and indications ofthe user's current input, such as the pointer on a screen or a stylus ona touchscreen.

In addition to causing a visual feedback delay, global optimizationprovides additional barriers to creating a natural appearance to acurvy, digital brush stroke. As an input path is being received from theuser, parameters are applied to the path for purposes of matching andsynthesizing a digital brush stroke. Application of parameters in corneror curved region of a curvy path line can yield distorted results whendone in the same manner as application of parameters in straightregions. With global optimization, however, the parameters in the cornerregions cannot be adjusted, thereby maintaining the artifacts anddistortion within the final synthesized brush stroke.

Embodiments of the present invention address the technical challenge ofperforming continuous pattern matching to create a curvy, digital brushstroke on the fly by creating series of temporary and permanent brushstroke segments from matching sets of received data samples to exemplarbrush stroke segments. As such, embodiments of the present disclosureaim to provide immediate or almost immediate visual feedback byperforming incremental local optimization with smaller amounts ofsamples without sacrificing the quality of the brush stroke generated.The disclosed methods for incrementally optimizing brush stroke segmentsprovide a level of expressive control and manual interaction to moreclosely mimic a physical painting experience.

Specifically, in accordance with embodiments of the disclosure, as datasamples indicating an input stroke are continuously received, sets ofsamples are detected from the data and matched to exemplar brush strokesegments. Based on the matched exemplar brush stroke segments, temporarybrush stroke segments are generated and displayed to provide quickfeedback to the user. When additional data samples are detected andmatched, new temporary segments are generated and displayed whileprevious temporary segments are replaced with permanent segments. Forexample, a first set of samples may be matched to a first set ofexemplar brush stroke segments, which is used to generate and display afirst temporary stroke segment. A second set of samples receivedimmediately after the first set of samples may then be matched, inconjunction with the first set, to second set of exemplar brush strokesegments. From the second set of exemplar brush stroke segments, a firstpermanent segment is generated to replace the first temporary segmentwhile a second temporary segment is also generated. The second temporarysegment is continuous with the first permanent segment and will besubsequently replaced by a permanent segment as more matching occurs.Local optimization through use of the permanent and temporary segmentstogether provide sufficient data to make intelligent matching decisionswhile allowing for brushing stroke synthesis on the fly.

Use of temporary segments that may be later updated to permanentsegments also allows for parameterization adjustments to accommodatecorner regions of a curvy brush stroke. As previously mentioned,parameterization of a corner region of an input stroke may beproblematic if done in the same manner as in straight regions.Specifically, during parameterization of an input stroke for purposes ofmatching, ribs may be applied perpendicular or substantiallyperpendicular to an input path, and when ribs intersect at cornerregions, undesired artifacts within the digital brush stroke may becreated. To address this problem, some configurations correctintersecting ribs as segments are updated from temporary segments topermanent segments. In particular, any intersecting rib may becollapsed, or adjusted so that the intersecting ribs all meet at asingle meeting point, and that meeting point is positioned at theendpoints of the intersecting ribs. Additionally, as ribs are collapsed,the rib lengths are compared to threshold values to detect distortion inthe corner regions. When distortion is detected, a brush stroke segmentis divided at a breaking sample, which has the maximum turning angle,and any collapsed ribs within the corner region of the segment arerestored to their pre-collapsed form. At least one corner structure isgenerated and inserted within the corner region. For example, a firstcorner structure may be generated based on the matching of the breakingsample, and a second corner structure may be generated based on thematching of a post-breaking sample. The two corner structures may beinserted at least between the breaking and the post-breaking samples toprovide a rounded, natural looking corner without shape or texturedistortion.

With reference now to the drawings, FIG. 1 is a block diagramillustrating an exemplary operating environment 100 for generatingcurved, digital brush strokes with continuous pattern matching inaccordance with implementations of the present disclosure. It should beunderstood that this and other arrangements described herein are setforth only as examples. Other arrangements and elements (e.g., machines,interfaces, functions, orders, and groupings of functions, etc.) can beused in addition to or instead of those shown, and some elements may beomitted altogether. Further, many of the elements described herein arefunctional entities that may be implemented as discrete or distributedcomponents or in conjunction with other components, and in any suitablecombination and location. Various functions described herein as beingperformed by one or more entities may be carried out by hardware,firmware, and/or software. For instance, various functions may becarried out by a processor executing instructions stored in memory.

The operating environment 100 is an example of a suitable architecturefor implementing certain aspects of the present disclosure. Among othercomponents not shown, the operating environment 100 includes a number ofuser devices, such as 102 a and 102 b, a network 104, storage 106, and abrush stroke synthesis engine 110. The brush stroke synthesis engineincludes a receiving module 112, a parameter module 114, a matchingmodule 120, a synthesis module 122, and a display module 124. Theparameter module 114 may further comprise a rib collapsing module 116and a corner breaking module 118.

It should be understood that the operating environment 100 shown in FIG.1 is an example of one suitable operating environment. Each of thecomponents shown in FIG. 1 may be implemented via any type of computingdevice, such as one or more of the computing device 1000, described inconnection to FIG. 10, for example. These components may communicatewith each other via the network 104, which may be wired, wireless, orboth. The network 104 can include multiple networks, or a network ofnetworks, but is shown in simple form so as not to obscure aspects ofthe present disclosure. By way of example, the network 104 can includeone or more wide area networks (WANs), one or more local area networks(LANs), one or more public networks, such as the Internet, and/or one ormore private networks. Where the network 104 includes a wirelesstelecommunications network, components such as a base station, acommunications tower, or even access points (as well as othercomponents) may provide wireless connectivity. Networking environmentsare commonplace in offices, enterprise-wide computer networks,intranets, and the Internet. Accordingly, the network 104 is notdescribed in significant detail.

It should be understood that any number of user devices, servers, anddata sources may be employed within the operating environment 100 withinthe scope of the present disclosure. Each of the above may comprise asingle device or multiple devices cooperating in a distributedenvironment. For instance, the brush stroke synthesis engine 110 may beprovided via multiple devices arranged in a distributed environment thatcollectively provide the functionality described herein. Additionally,other components not shown may also be included within the distributedenvironment. In some cases, functionality of the brush stroke synthesisengine 110 can be integrated, at least partially, into a user device,such as user devices 102 a or 102 b. Furthermore, the brush strokesynthesis engine 110 may at least partially be a cloud computingservice. The brush stroke synthesis engine 110 may be provided for by animage editing or graphic design software such as Adobe® Photoshop® orAdobe® Illustrator®.

Embodiments of the disclosure described herein may be implemented in astandalone painting simulation application or in a painting simulationmodule of a digital painting application or another type of graphicsapplication. For example, a digital painting application may present auser interface for receiving input from a user specifying values of oneor more criteria to be applied in digital painting operations, may modelpaint loaded on a virtual brush, may deposit paint onto a digitalcanvas. The brush stroke synthesis engine 110 operating within theoperating environment 100 may interface with such a digital paintingapplication to provide one or more methods for generating brush strokesbased on the user input.

The user devices 102 a and 102 b comprise any type of computing devicecapable of being operated by a user. In some implementations, the userdevices 102 a and 102 b are the type of computing devices described inrelation to FIG. 10 herein. By way of example and not limitation, a userdevice may be embodied as a personal computer (PC), a laptop computer, amobile device, a smartphone, a tablet computer, a smart watch, awearable computer, a personal digital assistant (PDA), an MP3 player, aglobal positioning system (GPS) or device, a video player, a handheldcommunications device, a gaming device or system, an entertainmentsystem, a vehicle computer system, an embedded system controller, aremote control, an appliance, a consumer electronic device, aworkstation, any combination of these delineated devices, or any othersuitable device.

The user devices 102 a and 102 b may include one or more processors, andone or more computer-readable media. The computer-readable mediaincludes computer-readable instructions executable by the one or moreprocessors. The instructions may be embodied by one or more applicationscapable of providing the brush stroke synthesis engine 110 to the user.It is contemplated herein that “application” be interpreted broadly. Asone example, an application can be installed on an operating system of auser device. As another example, an application can be a serviceintegrated into an operating system of a user device. In variousimplementations, an application can be on a user device but can at leastpartially be hosted server-side.

The user devices 102 a and 102 b, as well as other components of theoperating environment 100, may be communicatively coupled to storage106. Storage 106 generally stores information including data, computerinstructions (e.g., software program instructions, routines, orservices), logic, profiles, and/or models used in embodiments. Forinstance, storage 106 may store rules, conditions, associations,classification models, algorithms, or other criteria to perform thedisclosed methods of generating a digital brush stroke. Storage may alsobe used to store, among other things, libraries of exemplar brush strokesegments used in accordance with embodiments of the disclosure,temporary and/or permanent brush stroke segments generated, filescreated having digital brush strokes, and user preferences. In anembodiment, storage 106 comprises a data store (or computer datamemory). Further, although depicted as a single data store component,storage 106 may be embodied as one or more data stores or may be part ofa cloud-based platform.

As previously mentioned, the brush stroke synthesis engine 110 includesa receiving module 112, a parameter module 114, a matching module 120, asynthesis module 122, and a display module 124. The receiving module 112receives data indicating an input stroke. The data is received from oneor more input devices, a cursor control device, including a mouse, atouch pad, a button, a stylus on a tablet, a track ball, or anothersuitable input device. For example, the input stroke may be determinedby actions taken from “mouse down” to “mouse up”, or from the “click” ofa cursor control mechanism to the “release” of that mechanism, indifferent embodiments. Additionally, the input data received by thereceiving module 112 may be touch input received from a touchscreen onuser devices 102 a and 102 b through a stylus, finger, or otherinstrument.

Data indicating the input stroke determines, among other things, thecurvature and location of the digital brush stroke on the digital canvasas desired by the user. The digital brush stroke may also be defined byone or more values of other criteria, indicating the physical appearanceand behavior of the brushstroke. For example, in some embodiments, thedigital brush stroke generated is defined by color, lightness,transparency, brush shape, brush size, brush texture, canvas texture,and the like. Values for these criteria may be pre-defined or may be setby the user. Data indicating these values are received by the receivingmodule 112 of the brush stroke synthesis engine 110 or by anothercomponent.

Based on at least the input stroke and, in some embodiments, values forother criteria, the parameter module 114 of the brush stroke synthesisengine 110 identifies parameters of the user stroke by applying at leastan input path line and ribs are applied. Briefly turning to FIG. 2,parameterization of an input stroke 200 is illustrated. An input pathline 202 is applied to the input stroke 200 to run parallel with theinput stroke 200. The input path line 202 may represent the center ofthe input stroke. Ribs 204 are spaced along and transect the input pathline 202. Each rib 204 includes a first portion 206 extending from afirst side of the input path line 202 and a second portion 208 extendingfrom a second side of the input path line 202. The first portion 206 andthe second portion 208 of each rib 204 include endpoints 210 and 212,respectively. These endpoints 210 and 212 are distal from the input pathline 202.

Initially, the ribs 204 may be perpendicular or substantiallyperpendicular to the input path line 202 but may be adjusted inaccordance with some embodiments of the disclosure. Specifically, asshown in FIG. 1, the parameter module 114 includes a rib collapsingmodule 116 and a corner breaking module 118. The rib collapsing module116 and the corner breaking module 118 are discussed in greater detailwith respect to FIGS. 5A-8, but briefly, these components are configuredto make adjustments to the initial parameterization that enablesynthesis of more natural appearing corner sections in a curved, digitalbrush stroke.

In some embodiments, other parameters that are determined include startand stop points, indicating the beginning and ending of a brush stroke.Parameters may further include a stroke outline, indicating the boundaryof the digital brush stroke. The stroke outline may align with thesilhouette of the input stroke or may include a margin to preserveboundary effects, such as feathering. This parameterization may be doneby the parameter module 114, for instance, in accordance with themethods described in Lu et al., RealBrush: Painting with Examples ofPhysical Media (2013) (hereinafter referred to as Lu).

As parameterization identifies ribs 204, each rib 204 defines a samplewithin the input stroke. Based on the determined parameters, thematching module 120 matches the samples to exemplar brush strokesegments to generate the digital brush stroke. The exemplar brush strokesegments may already include the texture of the user's desired media,such as oil paint or acrylic paint, that will be applied to the inputstroke to create the digital brush stroke. The exemplar brush strokesegments may be part of a library of exemplar brush stroke segmentsstored within storage, such as storage 106. The exemplar brush strokesegments may be pre-loaded within the library or may be imported by theuser. For example, users may build their own libraries of exemplar brushstroke segments by capturing and uploading images of physical brushstrokes using various media and techniques. In some aspects, theseexemplar brush stroke samples are parameterized in the same manner asthe input stroke 200 to allow for matching of the samples of the inputstroke 200 to exemplar brush stroke segments. The matching module 120matches the samples on an individual sample basis or as sets of samples.Additionally, in some embodiments, the matching module 120 performsmatching upon a trigger such as, for instance, receiving a certainthreshold number of samples. Alternatively, the matching module 120performs matching continuously as each sample is received.

As matching occurs, the synthesis module 122 of the brush strokesynthesis engine 110 uses the exemplar brush stroke segments determinedto be best matches for the samples to synthesize a digital brush strokein accordance with the parameters of the input stroke and any othercriteria received for the digital brush stroke. Synthesis may include,at least, warping the exemplar brush stroke segments to fit the samplesto which they match and merging adjacent strokes using blendingtechniques. In some embodiments, the synthesis done by the synthesismodule 122 further includes considerations for interactions betweenbrush strokes when the user provides an input stroke that at leastpartially overlaps with a previously generated brush stroke. Examplematching and synthesis techniques that may be used in connection withthe disclosed embodiments include those techniques described in Lu.

Upon synthesis of the digital brush stroke, the digital brush stroke isdisplayed to the user by the display module 124. The display module 124may display the digital brush stroke on one or more display components,such as presentation components 1016 of the computing device 1000 inFIG. 10. The display module 124 generally displays segments of thedigital brush stroke as the segments are synthesized. In other words,the segments are displayed in real time or near real time.

As previously mentioned, the greater number of samples available formatching generally provides for greater accuracy in matching the samplesto exemplar brush stroke segments, producing a more natural appearancefor the digital brush stroke. However, waiting for more samples formatching and synthesis creates an undesirable delay in providing visualfeedback to the user. As such, embodiments of the present disclosureperform localized optimization with matching and brush stroke synthesisfor small sets of samples. To provide greater accuracy in matching withsmaller sized sample sets, the matching module 120 performs matching forsamples multiple times. Specifically, the matching module 120 performsmatching for a set of samples a first time and then performs matchingfor that set of samples a second time in conjunction with matching forsubsequently received samples. Based on the exemplar brush strokesegments matched in the first instance, the synthesis module 122generates a temporary segment, which is displayed to the user by thedisplay module 124. The synthesis module 122 then generates a permanentsegment to replace the temporary segment based on the second matching,which is determined from more samples. In example embodiments, once thepermanent segment is optimized, its appearance is finalized andimmutable until the drawing is finished. In contrast, the optimizationon the temporary segment gives an initial appearance but becomesre-matched and re-rendered at the next time optimization occurs. Byreplacing the initial temporary segment with a permanent segment havinga finalized appearance, instead of with another temporary segment, theuser is given a greater degree of certainty in the appearance of theportion of the brush stroke already synthesized.

Using permanent and temporary segments allows for local optimizationthat provides the user with visual feedback in real time or near realtime while providing enough information on the input stroke for thebrush stroke synthesis engine 110 to make intelligent matching andsynthesis decisions. The table provided below illustrates the concept ofusing temporary and permanent segments. Each row in the tablecorresponds to a point in time at which an optimization occurs, and eachcolumn corresponds to a sample along an input stroke.

Sample Index 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 t = 1 T1 T2 t = 2 P1 P2T1 T2 t = 3 P1 P2 T1 T2 t = 4 P1 P2 T1 T2 t = 5 P1 P2

The table above indicates the locations of permanent and temporarysegments for each time optimization is triggered. T1 and T2 indicate thestart and end, respectively, of a temporary segment, and P1 and P2indicate the start and end, respectively of the latest permanent segmentgenerated. In this example, there are fifteen samples within the inputstroke, and optimization is triggered five times. When drawing firststarts, at t=1, the entire beginning of the rendered brush stroke is atemporary segment. This temporary segment is based on a first set ofsamples, samples 1, 2, and 3. At t=2, a second set of samples,comprising samples 4, 5, and 6, are processed in conjunction with thefirst set of samples. The temporary segment in the previous timeinterval now becomes a permanent segment while a new temporary segmentis generated for the second set of samples. Continuing to t=3, a thirdset of samples, samples 7, 8, and 9, has been received and is processedwith the second set of samples. The previous temporary segmentrepresenting samples 4, 5, and 6 is replaced with a new permanentsegment, and a new temporary segment is generated for the third set ofsamples.

Continuing, at t=4, a fourth set of samples comprising samples 10, 11,and 12, are received and processed in conjunction with the third set ofsamples. A permanent segment is created to replace the temporary segmentfor the third set of samples generated at t=3, and a new temporarysegment is generated for the fourth set of samples. Finally, at t=5, afifth set of samples comprising samples 13, 14, and 15 are received andprocessed in conjunction with the fourth set of samples. The fifth setof samples represents the end of the user's input stroke, meaning nomore samples will be received for this brush stroke. As with previoussamples sets, a new permanent segment is generated to replace theprevious temporary segment for the fourth set of samples. Additionally,because no more input will be received for adjusting a temporary stroke,the fifth set of samples are represented in the permanent stroke,without the creation of an initial temporary segment. In other words,when the entire brush stroke is finished, the samples left to besynthesized become a permanent segment.

It is noted that the above table indicates only the permanent segmentrendered for that specified time interval. As such, though the tabledoes not explicitly indicate a permanent segment existing for samples 1,2, and 3 at t=3, the permanent segment generated for samples 1, 2, and 3at t=2 would still exist and be displayed to the user. Additionally,though the table above and this disclosure refer to discrete temporaryand permanent segments, each temporary segment is continuous with theabutting permanent section. For instance, the temporary segment forsamples 4, 5, and 6 is displayed as continuous with the permanentsegment for samples 1, 2, and 3.

In this example, optimization is not triggered by receipt of each samplebut, instead, is triggered when a threshold number of samples arereceived. Use of a threshold number of samples for triggeringoptimization represents a balance between visual delay and superiormatching and brush stroke synthesis. If the number of samples used totrigger optimization is too large, there is a perceivable visual delay.But if the number of samples is too small, matching and brush strokesynthesis will yield inferior results for the rendered digital brushstrokes.

The threshold number of samples depends on sample spacing betweensamples. If sample spacing is smaller, a larger threshold number ofsamples may be tolerable before the user is limited by visual delay. Butif the sample spacing is large, a smaller threshold value may be moreappropriate. Further, the threshold number may also vary based on themedia used as different media would behave differently when used forphysical paintings. For instance, wet media, such as watercolor, issubject to greater change in appearance for a period of time after aphysical brush stroke is painted, while a user would expect little to nochange to a physical brush stroke when using dry media, such as pencilor pastel. Accordingly, in some embodiments, the threshold value is setto a greater number when wet media is selected compared to dry mediabecause uncertainty in the appearance for a greater amount of time maybe expected and tolerated. In the example illustrated in the abovetable, the threshold number of samples for triggering matching andsynthesis is three; however, it is contemplated that other numbers ofsamples may be used for the threshold value.

After an optimization occurs, the user continues to input more samplesbefore the threshold number of samples is reached to trigger the nextoptimization. Some aspects mitigate the visual delay during the time thethreshold number of samples are being received by estimating futuretemporary segments based on the matching that occurred during theprevious optimization. In some embodiments, the last sample received forthe last optimization is incrementally extended for samples received butnot yet optimized. For example, referencing the table above, as a userinputs samples 7 and 8 during the time between t=2 and t=3, thosesamples are not yet matched because optimization will not be triggereduntil the ninth sample is received. In some embodiments, the exemplarbrush stroke segment matched to sample 6 by the matching module 120 isused to provide a rough estimate of exemplar brush stroke segments forsamples 7 and 8 before the next optimization is triggered. For example,if sample 6 was matched with the 10th sample of exemplar brush strokesegment 5 from an exemplar brush stroke segment library, it may beestimated that the next two samples from exemplar brush stroke segment 5(i.e., the 11th and 12th samples) provide a match for samples 7 and 8,respectively. As such, the display module 124 may display an estimatedtemporary segment continuous with the previous temporary segment createdat t=2 using the 11th and 12th samples of the exemplar brush strokesegment 5. When sample 9 is subsequently received, optimization is thentriggered for samples 7, 8, and 9 to determine the correct matching andappearance for those samples. The estimated segment generated forsamples 7 and 8 is replaced by the temporary segment.

Tuning to FIG. 3, a flow diagram depicting a method 300 for generating adigital brush stroke with continuous pattern matching in accordance withembodiments of the disclosure is provided. Each block of the method 300and any other methods described herein comprises a computing processperformed using any combination of hardware, firmware, and/or software.For instance, various functions can be carried out by a processorexecuting instructions stored in memory. The methods can also beembodied as computer-usable instructions stored on computer storagemedia. The methods can be provided by a standalone application, aservice or hosted service (standalone or in combination with anotherhosted service), or a plug-in to another product, to name a few. Themethod 300 may be performed at least in part, for instance, by the brushstroke synthesis engine 110 of FIG. 1.

Initially, as shown in block 302, data indicating an input stroke iscontinuously received. This data may be received by the receiving module112 from FIG. 1 through the use of one or more input devices. Forinstance, a user may be providing data indicating an input stroke bymoving a stylus along a touchscreen of a computing device. This data iscontinuously received contemporaneously as the user moving the stylus.Other data may also be received, including data relating to variouscriteria for the desired brush stroke such as color, texture, brushsize, and the like.

As the data is being continuously received, a first set of samples isdetected from the data, as shown in block 304. These samples may beportions of the input stroke between adjacent ribs. The first set ofsamples may comprise a threshold number of samples needed to triggeroptimization. If the threshold number of samples is five, for instance,the method would include detecting the first five samples within theinput stroke.

A first set of exemplar brush stroke segments are determined to be abest match with the first set of samples, as shown in block 306.Determining a first set of exemplar brush stroke segments is a bestmatch for the first set of samples may include the matching module 120from FIG. 1 comparing data indicating the first set of samples of theinput stroke to a plurality of exemplar brush stroke segments. Theexemplar brush stroke segments may be saved as a library of exemplarbrush stroke segments. Once a match is determined, a first temporarysegment is generated from the first set of exemplar brush strokesegments, as indicated by block 308. This may be performed by thesynthesis module 122 in FIG. 1. The first temporary segment is thendisplayed to the user, as shown in block 310.

The method 300 further includes, as shown in block 312, detecting asecond set of samples from the data indicating an input stroke. Thissecond set of samples may be detected in the same manner as the firstset of samples. The second set of samples may be continuous with thefirst set. In other words, the second set of samples is receivedimmediately after the first set of samples is received. In someembodiments, this second set of samples comprises the same number ofsamples as those detected in the first set, such as a threshold number.In alternative embodiments, however, the second set of samples includesa different number of samples than the first set.

Based on at least the first set of samples and the second set ofsamples, a second set of exemplar brush stroke segments is determined tobe a best match for the first set of samples and the second set ofsamples, as indicated at block 314. This matching may be done in asimilar manner to the matching of the first set of samples. A firstpermanent segment and a second temporary segment are generated from thesecond set of exemplar brush stroke segments, as shown in block 316.This step may be similar to the generation of the first temporarysegment except that two segments—the first permanent segment and thesecond temporary segment—are generated from this second set of samples.The first permanent segment and the second temporary segment are thendisplayed, as shown in block 318. The first permanent segment replacesthe first temporary segment and is displayed with the second temporarysegment as a continuous brush stroke. The second temporary segment maybe subject to change during a following optimization while the firstpermanent segment has a final appearance that will not change merely asresult of subsequent optimizations. This method 300 may continue foradditional sets of samples. For instance, a third set of samples may bedetected and used to generate a second permanent segment to replace thesecond temporary segment and a third temporary segment.

This method of utilizing incremental pattern matching and brush strokesynthesis with temporary and permanent segments allows for betterhandling of curvy strokes because incremental adjustments may be made tocorrect issues with corner segments of the brush stroke. As previouslymentioned, during initial parameterization, an input path line and ribsare applied to determine the parameters of the input stroke. Thisparameterization, however, can lead to artifacts being rendered incorners of a curvy brush stroke. FIG. 4A displays a rendered brushstroke 400 having such an artifact 402. The artifact 402 is caused byintersecting ribs being used during the matching and synthesis process.

To correct this issue, embodiments of the parameter module 114 in FIG. 1includes a rib collapsing module 116 to adjust the ribs upon detectionof intersecting ribs. Additionally, as discussed in greater detail belowwith respect to FIGS. 7A-B, the parameter module 114 includes a cornerbreaking module 118 to correct distortion that sometimes results fromrib collapsing. Together, the rib collapsing module 116 and the cornerbreaking module 118 provide a rounded, natural-looking corner of a curvewithout artifacts resulting from traditional parameterization, which isshown by the corrected brush stroke 410 in FIG. 4B.

Focusing on the rib collapsing module 116, FIGS. 5A-5C illustrate howthe ribs are adjusted in accordance with embodiments of the disclosure.In FIG. 5A, an input stroke 500 with an input path line 502 and ribs 504are shown. Each rib 504 includes a first portion 506 on one side of theinput path line 502 and a second portion 508 on the other side. Assamples are detected from the received data and parameters aredetermined with the addition of ribs 504, the rib collapsing module 116detects intersecting ribs. Specifically, the rib collapsing module 116may determine whether a rib intersects with an immediately adjacent rib.If the newly added rib does not intersect with the adjacent rib, it isnot adjusted. If the ribs do intersect, any rib within the temporarysegment that intersects may be collapsed. As used herein, “collapsed”means adjusted such that the respective endpoints of the ribs meet orcome together at a single meeting point, which is the only place wherethe ribs meet. In other words, the collapsed ribs come together at ameeting point positioned at their respective endpoints, rather thantransecting one another. The adjustments that are made may vary and maycomprise increasing rib length, decreasing rib length, increasing theangle between the rib and the input path line, and/or decreasing theangle between the rib and the input path line. Adjustments may be madeto the portion of the rib that intersects another rib while the portionof the rib not involved in intersection may remain unmodified. As theadjusted, intersecting portion will be on an interior side of a corner,the adjusted portion may be referred to as an interior section, and theunadjusted portion may be the exterior portion. In some aspects, whendetermining which ribs to collapsed, only samples after a previouscorner section are considered so that adjustments made for a new cornersection do not interfere with the shape of a previous corner.

Looking to FIG. 5A, there are ribs 504A-G that have already beencollapsed to meet at a single meeting point 514. The rib collapsingmodule 116 determines that a new rib 504H intersects an adjacent rib504G as well as ribs 504A-F, triggering additional rib collapsing asshown in FIG. 5B. In FIG. 5B, the new rib 504H no longer intersects ribs504A-G. Instead, new rib 504H has been adjusted so that its endpoint 512meets the endpoints of ribs 504A-G at meeting point 514. Only the firstportion 506 (i.e., the interior portion) of the new rib 504H thatintersected with other ribs was adjusted while the second portion 508(i.e., the exterior portion) of the new rib 504H was not adjusted.

This process continues as a corner section continues to be created. FIG.5C illustrates input stroke 500 after several more samples have beenreceived. As the rib collapsing module 116 determined that the ribs504I-Q intersected with at least an adjacent rib, each rib 504I-Q wascollapsed such that the endpoints meet at the single meeting point 514.As seen by comparing the meeting point 514 in FIGS. 5B-C, the locationof the meeting point 514 may be determined by all the ribs that meet atthe meeting point 514 and, therefore, may move as the additional ribs504I-Q are collapsed. Additionally, when the meeting point 514 moves,the previously collapsed ribs 504A-H may be collapsed again and adjustedin a way so that the endpoints of ribs 504A-H continue to meet at themeeting point 514. Further, there may be times, particularly with sharpcorners, when newly added ribs intersect ribs that were not previouslyintersecting. For instance, ribs 504R-S in FIG. 5C were added prior toribs 504A-H but were not were not previously intersecting ribs in FIG.5A-B. However, one or more of the additional ribs 504I-Q may haveintersected with ribs 504R-S, causing ribs 504R-S to be collapsed aswell.

FIG. 6 provides a flow diagram illustrating the method 600 of handlingcorner sections with rib collapsing in accordance with embodiments ofthe disclosure. At block 602, data indicating an input stroke iscontinuously received. This step may be done by the receiving module 112from FIG. 1. As the data is received, parameters of the input stroke aredetermined, as indicated at block 604. The parameters may be determinedby an input path line that runs parallel to the input stroke andrepresenting the center of the input stroke and ribs transecting theinput path line. Each rib may comprise at least one endpoint distal tothe input path line.

After the initial parameterization, a first rib is detected asintersecting with at least a second rib, which is provided in block 606.The first rib may be adjacent to the second rib and may immediatelyprecede the second rib. One or more of the intersecting ribs areadjusted so that the ribs no longer intersect, as shown in block 608.For instance, the first rib and/or the second rib may each be adjustedso that the endpoint of the first rib and the endpoint of the second ribcome together at a first meeting point. This first meeting point may bethe only point at which the first rib and the second rib meet. Thedetection of intersecting ribs and the adjustments may be performed bythe rib collapsing module 116 of FIG. 1.

As indicated at block 610, a curved, digital brush stroke is generatedbased on the ribs and the input path line. The generation of the brushstroke may be done through matching samples to exemplar brush strokesegments and synthesizing the digital brush stroke from the matchedexemplar brush stroke segments, as described with respect to FIGS. 1-3.

Though not shown FIG. 6, the method may further include collapsingadditional ribs. For instance, the method may comprise detecting a thirdrib intersects with the second rib, the third rib being adjacent to thesecond rib. At least one of the intersecting ribs, such as the thirdrib, may be adjusted so that the endpoint of the third rib meets withendpoints of the first and second rib at the meeting point. The meetingpoint may be moved from a first location after the initial adjustment ofthe second rib to a second location after the adjustment of the thirdrib. The first rib and/or the second rib may also be adjusted to allowfor movement of the meeting point.

By utilizing rib collapsing in accordance with embodiments of thisdisclosure, the curved section of the digital brush stroke, as shown byFIG. 4B, does not have artifacts. However, the adjustments to the ribsmay result in noticeable texture stretching and/or shape distortion.This problem typically occurs when the corner has a sharp turning angle.To mitigate the stretching and distortion, some embodiments of theparameter module 114 in FIG. 1 also includes a corner breaking module118. The corner breaking module 118 detects when distortion occurs.Detection may occur after ribs are collapsed and when new ribs are nolonger intersecting with other ribs. In other words, detection may occurupon the completion of a corner section.

To detect corner distortion, the corner breaking module 118 determineswhether a length of at least a portion of a rib is outside of one ormore threshold values. For example, a first portion of a rib may have alength between the input path line and an endpoint point distal to theinput path line. The length may be changed when the first portion of therib is adjusted during rib collapsing so that the endpoint of the ribmeets with other intersecting ribs at a single meeting point. Distortionmay be detected when it is determined that the length of the rib in thecollapsed form is either too great or too small relative to the lengthof the rib non-collapsed form, i.e., the length of the rib before ribcollapsing occurred. When distortion is detected, the corner breakingmodule 118 scans through previous samples in the corner section toidentify a breaking sample and a post-breaking sample. The breakingsample is a sample having the greatest turning angle between twoexterior portions of ribs, and the post-breaking sample is a sampleimmediately following the breaking sample. The brush stroke is dividedat the breaking sample into a pre-break segment and a post-breaksegment. The corner breaking module 118 inserts corner structuresbetween the pre-break and post-break segments by extending the breakingsample in a forward direction and extending the post-breaking sample ina backward direction. The corner structures may help provide a roundedand natural-looking corner without shape or texture distortion. In someembodiments, previously collapsed ribs are restored to non-collapsedform before corner structures are inserted. FIGS. 7A-B illustrate thecorner breaking processes performed by the corner breaking module 118.

FIG. 7A depicts an input stroke 700 after rib collapsing. The inputstroke 700 has an input path line 702 and ribs 704. Each rib 704comprises a first portion 706 on one side of the input path line 702 anda second portion 708 on the opposite side of the input path line 702.Because the input stroke 700 comprises a corner section, the firstportions 706 may also be referred to as interior portions, and thesecond portions 708 may also be referred to as exterior portions. Eachfirst portion 706 comprises a length between the input path line 702 andan endpoint 712 of the first portion 706, the endpoint 712 being distalto the input path line 702.

As ribs are collapsed by the rib collapsing module 116, the cornerbreaking module 118 looks for corner distortion by determining whetherthe length of each first portion 706 of a collapsed rib 704 is outsideone or more threshold values. The threshold values may be determined bythe length of the first portion 706 in a non-collapsed form. Forexample, in some configurations, distortion is determined to be presentwhere the length in collapsed form is either greater than or equal to150% of the length in non-collapsed form or less than 50% the length innon-collapsed form. Different threshold values may be used in otherembodiments. In some embodiments, the rib collapsing module 116considers the length of each first portion 706 of a collapsed rib 704 todetermine whether the length falls outside of the threshold values. Inother aspects, distortion is detected for a segment by looking only atthe length of the shortest first portion 706, which is shown as arrow716 in FIG. 7A, and the length of the longest first portion 706,indicated by arrow 718. Further, in some embodiments, the cornerbreaking module 118 detects distortion as each segment is optimized, andin other aspects, distortion is detected only when the new rib stopsintersecting with previous rib.

When distortion is detected, the corner breaking module 118 identifiesthe breaking sample, which is used to break the stroke into a post-breaksegment and a pre-break segment. The breaking sample may be the samplewith the greatest turning angle. In FIG. 7A, the breaking sample 720 isfound to have the greatest turning angle 724. The sample immediatelyfollowing the breaking sample 720 is referred to as the post-breakingsample 722. Input stroke 700 is broke into two segments at the breakingsample 720. The pre-break segment 726 may comprise samples from thebeginning of the current segment to the breaking sample 720, and thepost-break segment 728 may comprise samples from the post-breakingsample 722 to the most recent sample 730.

As shown in FIG. 7B, the corner breaking module 118 restores previouslycollapsed ribs into a non-collapsed form. When restoring ribs 704, onlysamples within the current corner section in which distortion isdetected may have ribs 704 restored to prevent interference withprevious portions of the brush stroke. Because the collapsed ribs 704had caused the distortion that the corner breaking module 118 seeks tofix using corner structures, restoring the ribs 704 is done to removethe distortion so that the corner structures are not based on the ribs704 in the distorted form. It may be noted, however, that not all theribs 704 in FIG. 7B appear as restored. Once corner structures areinserted to fix the distortion within the corner, some of the ribs 704may be collapsed again if necessary to remove artifacts, which is whathas happened in FIG. 7B.

After ribs are restored, the corner breaking module 118 inserts cornerstructures between the pre-break segment 726 and the post-break segment728. The corner structures in FIG. 7B are depicted in dashed lines. Afirst corner structure 732 may be created by extending the breakingsample 720 forward along a tangential direction and inserting verticesin that direction to form a semicircular structure, shown by thesubstantially horizontal dashed lines in FIG. 7B. To extend “forward,”as used herein, means to extend towards more recent samples in thedirection the user is drawing the input stroke 700, which is shown byarrow 736 in FIG. 7B. As such, the first corner structure 732 isextended from the breaking sample 720 toward the post-breaking sample722. In some embodiments, such as the one shown in FIG. 7B, the firstcorner structure extends beyond the post-breaking sample 722. Thetangential direction for the first corner structure 732 is determined bythe tangential line based on the breaking sample 720 and samples beforethe breaking sample 720. For example, if the breaking sample 720 is “i”and the post-breaking sample is “i+1”, then the tangent line for thefirst corner structure 732 may be calculated as P_i−P_i−2.

Similarly, the second corner structure 734 is created by extending thepost-breaking sample 722 backward along the tangential direction andinserting vertices in that direction to form another semicircularstructure, shown by the substantially vertical dashed lines in FIG. 7B.As used herein, extending “backward” means to extend towards earliersamples in a direction opposite of the direction that the user isdrawing the input stroke 700, which is shown by arrow 738 in FIG. 7B. Inother words, the second corner structure 734 extends from thepost-breaking sample 722 to beyond the breaking sample 720. Thetangential direction for the second corner structure 734 is determinedby the tangential line calculated from the post-breaking sample 722 andsamples after the post-breaking sample 722. For instance, if thebreaking sample 720 is “i” and the post-breaking sample is “i+1”, thenthe tangent line for the second corner structure 734 may be calculatedas P_i+3−P_i+1.

Because the first corner structure 732 extends forward from the breakingsample 720 and the second corner structure 734 extends backward from thepost-breaking sample 722, the first corner structure 732 and the secondcorner structure 734 may overlap between the breaking sample 720 and thepost-breaking sample 722. In some embodiments, the first cornerstructure 732 and the second corner structure 734 are perpendicular toeach other.

As previously mentioned, in some embodiments, the ribs 704 are collapsedas needed. As shown in FIG. 7B, some ribs 704 in the post-break segment728 are collapsed while other ribs 704, including all the ribs 704 inthe pre-break segment 726, are left in their restored, non-collapsedforms. The ribs 704 in the post-break segment 728 that have beenre-collapsed now converge toward a new meeting point 740. The newmeeting point 740 is in a different location than the meeting point 714from the initial collapsing because not all the ribs 704 that werepreviously collapsed and meeting at the meeting point 714 are beingcollapsed again. As with the initial collapsing, ribs 704 will start tobe collapsed when the rib 704 intersects with an immediately adjacentrib, as explained above with respect to FIGS. 5A-C.

Some brush strokes, however, will not require any ribs to be restoredafter the corner structures are inserted. Turning briefly to FIG. 9, aninput stroke 900 after corner structures 908 and 910 are inserted isprovided. This input stroke 900 has a very sharp turning angle and, assuch, no ribs 902 on either the pre-breaking segment 904 or thepost-breaking segment 906 are re-collapsed. While restored ribs 902 inthe pre-break segment 904 intersect with ribs 902 in the post-breaksegment 906, ribs 904 do not intersect other ribs 904 within the samesegment and, therefore, are not collapsed again.

When corner structures are inserted and ribs re-collapsed, if needed,brush stroke synthesis may occur based on texture coordinates. Tosynthesize the portion of the brush stroke for the corner structures,the texture coordinates are derived from the coordinates of knownsamples. Specifically, turning back to FIG. 7, the texture coordinatesfor the first corner structure 732 are derived from the texturecoordinate at the breaking sample 720, and texture coordinates for thesecond corner structure 734 are derived from the post-breaking sample722. The first corner structure 732 may be matched to samples innumerical order beginning with the sample immediately following thesample to which the breaking samples was matched. For instance, if thebreaking sample 720 is matched to the 10th sample of exemplar brushstroke segment 5, samples within the first corner structure 732 may bematched to the 11th, 12th, 13th, to the Nth sample in the exemplar brushstroke segment 5, depending on how many samples are within the firstcorner structure 732. Matching for the second corner structure 734, onthe other hand, may be in reverse numerical order beginning with theexemplar brush stroke segment sample immediately before the sample towhich the post-breaking sample matches. For example, if thepost-breaking sample 722 is matched to the 11th sample of the exemplarbrush stroke 5, samples within the second corner structure 734 may bematched to the 10th, 9th, and 8th sample in the exemplar brush strokesegment 5. The first corner structure 732 and the second cornerstructure 734 together create a rounded and natural looking cornerwithout shape distortion or texture stretching.

It is noted that while the examples for matching samples provided abovematched samples in increments or decrements of 1, the increment ordecrement used in matching depends on the sampling rates used for thefirst and second corner structures 732 and 734, respectively. Highersample rates for the first and second corner structures 732 and 734,respectively, cause the matching to occur in increments and decrementsless than 1.

A method 800 of creating a digital, curved brush stroke with cornerbreaking in accordance with embodiments of the disclosure is depicted inthe flow diagram provided in FIG. 8. As shown in block 802, dataindicating an input stroke is received by, for instance, the receivingmodule 112 of FIG. 1. Initial parameters may be applied to the inputstroke. Specifically, the input stroke may include an input path linethat represents a line in the center of the input stroke and runningparallel to the input stroke. The input stroke may further include ribstransecting the input path line. Each rib may comprise a first portionextending along a first side of the input path line and a second portionextending along a second side of the input path line. Each first portionmay comprise a length between the input path line and an endpoint of thefirst point that is distal to the input path line.

As indicated by block 804, the method further includes detecting cornerdistortion by determining the length of at least a portion of a rib,such as the first portion, is outside one or more threshold values. Thisstep may be performed by the corner breaking module 118 of FIG. 1.Detecting corner distortion occurs after intersecting ribs have beencollapsed by the rib collapsing module 116 of FIG. 1. As such,determining the length of at least a portion of the rib is outside oneor more threshold values may include determining the length of the firstportion of the rib in a collapsed form is outside threshold values basedon the length of the first portion of the rib in a non-collapsed form.For example, in some embodiments, distortion is detected when the lengthof first portion in collapsed form either greater than or equal to 150%of the length of first portion in non-collapsed form or is less than 50%of the length of the first portion in non-collapsed form.

Upon detecting corner distortion is present, a breaking sample and apost-breaking sample are identified, as shown in block 806. The breakingsample is a sample with the greatest turning angle, or the greatestangle along the input path at a sample. The post-breaking sample is asample immediately following the breaking sample. The input stroke isdivided at the breaking sample into a pre-break and a post-breaksegment. At block 808, at least one corner structure is inserted withinthe corner region. In some aspects, a first corner structure and asecond corner structure are inserted between the breaking sample and thepost-breaking sample. The first corner structure may comprise firstvertices extending from the breaking sample to at least thepost-breaking sample, and the second corner structure may comprisesecond vertices extending from the post-breaking sample to at least thebreaking sample. In some aspects, the first corner structure continuesto at least one or more samples immediately following the post-breakingsample, and the second corner structure continues to at least one ormore samples immediately preceding the breaking sample.

Inserting the first corner structure and the second corner structure mayinclude re-adjusting one or more intersecting ribs into non-collapsedform, inserting the first vertices using texture coordinates of thebreaking sample, and inserting the second vertices using texturecoordinates of the post-breaking sample. The first vertices may extendin a first direction corresponding to a line tangential to the breakingsample while the second vertices extend in a second direction correspondto a line tangential to the post-breaking sample.

A curved, digital brush stroke is generated based on at least the ribs,the input path line, and the at least one corner structure, such as thefirst and second corner structures, as shown in block 810. Thegeneration of the brush stroke may be done by matching texturecoordinates of the samples to exemplar brush stroke segments andsynthesizing a digital brush stroke from the exemplar brush strokesegments and the parameters of the input stroke. The samples within thefirst corner structure may be matched to samples of the same exemplarbrush stroke segment to which the breaking sample is matched and,specifically, may be matched to samples sequentially following thesample of the exemplar brush stroke segment to which the breaking sampleis matched. Similarly, the samples within the second corner structuremay be matched to samples of the same exemplar brush stroke segment towhich the post-breaking sample is matched and may be matched to samplespreceding the sample to which the post-breaking sample is matched inreverse numerical order.

Having described embodiments of the present invention, an exemplaryoperating environment in which embodiments of the present invention maybe implemented is described below in order to provide a general contextfor various aspects of the present invention. Referring initially toFIG. 10 in particular, an exemplary operating environment forimplementing embodiments of the present invention is shown anddesignated generally as computing device 1000. Computing device 1000 isbut one example of a suitable computing environment and is not intendedto suggest any limitation as to the scope of use or functionality of theinvention. Neither should the computing device 1000 be interpreted ashaving any dependency or requirement relating to any one or combinationof components illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Theinvention may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The invention may alsobe practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

With reference to FIG. 10, computing device 1000 includes a bus 1010that directly or indirectly couples the following devices: memory 1012,one or more processors 1014, one or more presentation components 1016,input/output (I/O) ports 1018, input/output components 1020, and anillustrative power supply 1022. Bus 1010 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 10 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component. Also, processors have memory. Thisdisclosure recognizes that such is the nature of the art, and it isreiterated that the diagram of FIG. 10 is merely illustrative of anexemplary computing device that can be used in connection with one ormore embodiments of the present invention. Distinction is not madebetween such categories as “workstation,” “server,” “laptop,” “hand-helddevice,” etc., as all are contemplated within the scope of FIG. 10 andreference to “computing device.”

Computing device 1000 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 1000 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computing device 1000. Computer storagemedia does not comprise signals per se. Communication media typicallyembodies computer-readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer-readable media.

Memory 1012 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 1000includes one or more processors that read data from various entitiessuch as memory 1012 or I/O components 1020. Presentation component(s)1016 present data indications to a user or other device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

I/O ports 1018 allow computing device 1000 to be logically coupled toother devices including I/O components 1020, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc. The I/Ocomponents 1020 may provide a natural user interface (NUI) thatprocesses air gestures, voice, or other physiological inputs generatedby a user. In some instance, inputs may be transmitted to an appropriatenetwork element for further processing. A NUI may implement anycombination of speech recognition, touch and stylus recognition, facialrecognition, biometric recognition, gesture recognition both on screenand adjacent to the screen, air gestures, head and eye tracking, andtouch recognition associated with displays on the computing device 1000.The computing device 1000 may be equipped with depth cameras, such as,stereoscopic camera systems, infrared camera systems, RGB camerasystems, and combinations of these for gesture detection andrecognition. Additionally, the computing device 1000 may be equippedwith accelerometers or gyroscopes that enable detection of motion. Theoutput of the accelerometers or gyroscopes may be provided to thedisplay of the computing device 1000 to render immersive augmentedreality or virtual reality.

As can be understood, embodiments of the present invention provide for,among other things, creating digital brush strokes using local,incremental optimization to reduce delay in visual feedback and tocorrect artifacts and distortions that arise in corner sections of thebrush strokes. The present invention has been described in relation toparticular embodiments, which are intended in all respects to beillustrative rather than restrictive. Alternative embodiments willbecome apparent to those of ordinary skill in the art to which thepresent invention pertains without departing from its scope.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects set forth above, togetherwith other advantages which are obvious and inherent to the system andmethod. It will be understood that certain features and subcombinationsare of utility and may be employed without reference to other featuresand subcombinations. This is contemplated by and is within the scope ofthe claims.

What is claimed is:
 1. One or more non-transitory computer-readablemedia having a plurality of executable instructions embodied thereon,which, when executed by one or more processors, cause the one or moreprocessors to perform a method for continuous pattern matching togenerate a digital brush stroke, the method comprising: generating fordisplay a first temporary segment from a first set of exemplar brushstroke segments selected based on a first set of samples from inputstroke data; and generating for display a first permanent segment and asecond temporary segment from a second set of exemplar brush strokesegments selected based on the first set of samples and a second set ofsamples from the input data, wherein generating for display the firstpermanent segment and the second temporary segment comprises generatingthe first permanent segment from one or more exemplar brush strokesegments selected based on the first set of samples and the second setof samples, wherein the first permanent segment replaces the firsttemporary segment when the first permanent segment and the secondtemporary segment are displayed.
 2. The computer-readable media of claim1, wherein the second set of samples is received after the first set ofsamples is received.
 3. The computer-readable media of claim 1, whereinthe method further comprises determining parameters of an input strokeby applying an input path line representing a center of the input strokeand a plurality of ribs transecting the input path line.
 4. Thecomputer-readable media of claim 3, wherein the method furthercomprises: prior to generating for display the first permanent segmentand the second temporary segment, detecting intersecting ribs from atleast a first rib intersecting with a second rib; and adjusting at leastone of the intersecting ribs such that the intersecting ribs no longerintersect, wherein the first permanent segment is based at least on theinput path line and the plurality of ribs after at least adjusting theat least one of the intersecting ribs.
 5. The computer-readable media ofclaim 4, wherein the method further comprises: after adjusting at leastone of the intersecting ribs, detecting corner distortion by determininga length of at least a portion of an intersecting rib is outside atleast one threshold value; and inserting at least one corner structurewithin a corner region of the input stroke, wherein the permanentsegment is further based on the at least one corner structure.
 6. Thecomputer-readable media of claim 1, wherein the method furthercomprises: generating a second permanent segment and a third temporarysegment based on at least the second set of samples and a third set ofsamples from the input stroke data, wherein the second permanent segmentreplaces the second temporary segment.
 7. The computer-readable media ofclaim 1, wherein generating for display the first temporary segmentcomprises determining the first set of exemplary brush stroke segmentsis a match for the first set of samples, and wherein generating fordisplay the first permanent segment and the second temporary segmentcomprises determining the second set of exemplary brush stroke segmentsis a match for the first set of samples and the second set of samples.8. One or more non-transitory computer-readable media having a pluralityof executable instructions embodied thereon, which, when executed by oneor more processors, cause the one or more processors to perform a methodfor generating a corner of a curved, digital brush stroke, the methodcomprising: continuously receiving data indicating an input stroke, thedata including an input path line representing a center of the inputstroke and a plurality of ribs transecting the input path line, each ribcomprising an interior portion inside of a curve formed by the inputpath line and an exterior portion outside of the curve formed by theinput path line; detecting intersecting ribs from at least a first ribintersecting with a second rib; adjusting at least one of theintersecting ribs such that the intersecting ribs no longer intersect,wherein adjusting at least one of the intersecting ribs comprisesadjusting a position of the interior portion of at least oneintersecting rib and not adjusting a position of the exterior portion ofthe at least one intersecting rib; and generating a curved, digitalbrush stroke based on the plurality of ribs and the input path line. 9.The one or more computer-readable media of claim 8, wherein the firstrib is adjacent to and immediately precedes the second rib.
 10. The oneor more computer-readable media of claim 8, wherein adjusting at leastone of the intersecting ribs such that the intersecting ribs no longerintersect comprises adjusting at least one of the intersecting ribs suchthat endpoints of the intersecting ribs meet at a first meeting point,wherein the endpoints are distal to the input path line.
 11. The one ormore computer-readable media of claim 10, wherein the method furthercomprises: detecting a third rib intersects with the second rib, thethird rib being adjacent the second rib; adjusting at least one of thefirst rib, the second rib, and the third rib such that the endpoints ofeach of the first rib, the second rib, and the third rib intersect at asecond meeting pointing point, wherein the second meeting point replacesthe first meeting point.
 12. The one or more computer-readable media ofclaim 8, wherein the curved brush stroke comprises a plurality ofsamples spaced along the input path line, and wherein generating thecurved, digital brush stroke is further based on matching each samplewithin the plurality of samples to an exemplar brush stroke segmentstored in an exemplar brush stroke segment library.
 13. The one or morecomputer-readable media of claim 8, wherein, prior to adjusting at leastone of the intersecting ribs such that the intersecting ribs no longerintersect, a temporary segment based on at least the intersecting ribsis generated, and wherein generating the curved, digital brush strokecomprises at least generating a permanent segment to replace thetemporary segment.
 14. A computer-implemented system comprising: one ormore processors; and one or more computer storage media storingcomputer-usable instructions to cause the one or more processors toperform operations comprising: receiving data indicating an inputstroke, the input stroke having parameters that include an input pathline representing a center of the input stroke and a plurality of ribstransecting the input path line; detecting corner distortion bydetermining a length of at least a portion of a rib is outside at leastone threshold value; inserting at least one corner structure within acorner region of the input stroke; and generating a curved brush strokebased on the plurality of ribs, the input path line, and the at leastone corner structure.
 15. The computer-implemented system of claim 14,wherein the method further comprises: prior to detecting cornerdistortion, adjusting two or more intersecting ribs from a non-collapsedform to a collapsed form in which the endpoints of each intersecting ribmeet at a single meeting point.
 16. The computer-implemented system ofclaim 15, wherein corner distortion is detected when the length of atleast a portion of a rib in the collapsed form is less than half of alength in the non-collapsed form or is greater than or equal to one andhalf times the length in the non-collapsed form.
 17. The system of claim15, wherein inserting at least one corner structure comprises insertingat least a first corner structure comprising a plurality of firstvertices and a second corner structure comprising a plurality of secondvertices.
 18. The system of claim 17, wherein the first corner structureextends from a breaking sample to at least a post-breaking sample andthe second corner structure extends from the post-breaking sample to atleast the breaking sample, wherein the breaking sample comprises asample having a greatest angle between two adjacent ribs prior toinserting the first corner structure and the second corner structure andwherein the post-breaking sample is a sample immediately following thebreaking sample.
 19. The computer-implemented system of claim 18,wherein inserting the first corner structure and the second cornerstructure comprises: re-adjusting each intersecting rib into thenon-collapsed form; inserting the plurality of first vertices in a firsttangential direction determined from at least a sample in an exemplarbrush stroke segment to which the breaking sample matches; and insertingthe plurality of second vertices in a second tangential directiondetermined from at least a sample in an exemplar brush stroke segment towhich the post-breaking sample matches.
 20. The computer-implementedsystem of claim 18, wherein generating the curved brush strokecomprises: matching the first corner structure to first samples of anexemplar brush stroke segment, the first samples being in numericalorder beginning with a sample immediately following a sample to whichthe breaking sample matches; and matching the second corner structure tosecond samples of an exemplar brush stroke segment, the second samplesbeing in reverse numerical order beginning with a sample immediatelybefore a sample to which the post-breaking sample matches.